In real-world classification tasks, each class often comprises multiple finer-grained subclasses. As the subclass labels are frequently unavailable, models trained using only the coarser-grained class labels often exhibit highly variable performance across different subclasses. This phenomenon, known as hidden stratification, has important consequences for models deployed in safety-critical applications such as medicine. We propose GEORGE, a method to both measure and mitigate hidden stratification even when subclass labels are unknown. We first observe that unlabeled subclasses are often separable in the feature space of deep models, and exploit this fact to estimate subclass labels for the training data via clustering techniques. We then use these approximate subclass labels as a form of noisy supervision in a distributionally robust optimization objective. We theoretically characterize the performance of GEORGE in terms of the worst-case generalization error across any subclass. We empirically validate GEORGE on a mix of real-world and benchmark image classification datasets, and show that our approach boosts worst-case subclass accuracy by up to 15 percentage points compared to standard training techniques, without requiring any information about the subclasses.

Strengths: This paper focus the hidden stratification problem of training deep neural networks with only coarse-grained class labels that results in variable performance across different subclasses. Motivated by the observation of feature representation of deep neural networks often capture information about unlabeled subclasses, this paper proposed GEORGE, a two-step method for mitigating hidden stratification. In the first step, GEORGE estimates subclass labels in feature space via Gaussian mixture model clustering. Then in the second step, the estimated subclass labels are used in a distributional robust optimization objecitve to train a robust classifier. The strengths of this work are: 1.This paper is overall complete.

This paper initially received scores of 7,7,7, and 5, and after the rebuttal R4 revised up from a 5 to a 6. The reviewers all commented in the discussion that they were satisfied with the responses they received in the rebuttal. The problem of discovering hidden unlabelled subclasses in labelled datasets is an interesting one and relevant to a general machine learning audience. The authors are encouraged to use the suggestions in the reviews to improve the clarity of the paper e.g. R1's comment about the clustering step, fix the typos (R2), and polish the figures (R3, R1).

In real-world classification tasks, each class often comprises multiple finer-grained "subclasses." As the subclass labels are frequently unavailable, models trained using only the coarser-grained class labels often exhibit highly variable performance across different subclasses. This phenomenon, known as hidden stratification, has important consequences for models deployed in safety-critical applications such as medicine. We propose GEORGE, a method to both measure and mitigate hidden stratification even when subclass labels are unknown. We first observe that unlabeled subclasses are often separable in the feature space of deep models, and exploit this fact to estimate subclass labels for the training data via clustering techniques.